Multi-class ROC analysis from a multi-objective optimisation perspective

The Receiver Operating Characteristic (ROC) has become a standard tool for the analysis and comparision of classifiers when the costs of misclassification are unknown. There has been relatively little work, however, examining ROC for more than two classes. Here we discuss and present a number of different extensions to the standard two-class ROC for multi-class problems. We define the ROC surface for the Q-class problem in terms of a multi-objective optimisation problem in which the goal is to simultaneously minimise the Q(Q − 1) misclassification rates, when the misclassification costs and parameters governing the classifier’s behaviour are unknown. We present an evolutionary algorithm to locate the Pareto front—the optimal tradeoff surface between misclassifications of different types. The performance of the evolutionary algorithm is illustrated on a synthetic three class problem, for both k-nearest neighbour and multi-layer perceptron classifiers. Neuroscale is used to visualise the 5-dimensional front in two or three dimensions. The use of the Pareto optimal surface to compare classifiers is discussed, together with Hand & Till’s [2001] M measure of total class separability. We present a straightforward multi-class analogue of the Gini index. Also, we develop an evolutionary algorithm for the maximisation of M for the situation in which the parameters of the classifier can be varied. This is illustrated on various standard machine learning data sets.